⬅ Back to Menu

What is Publication Bias?

Addressing Publication Bias in Meta-Analyses

In meta-analyses, we can apply techniques to partially mitigate the risk of distortions caused by publication and reporting bias. Some of these strategies focus on improving the study search process, while others involve statistical adjustments.

  1. Study Search - Several specific approaches can help minimize the risk of bias:

    • Include Grey Literature: such as dissertations, preprints, government reports, and conference proceedings;

    • Backward and Forward Snowballing: Review the references of included studies (backward snowballing) and identify papers that have cited those studies (forward snowballing).

    • Search in Multiple Databases

    • Comprehensive Search Strategies

    • Contacting Study Authors for Unpublished Data

  2. Statistical methods: Statistical methods can help detect signs of publication bias by analyzing patterns in the data, such as asymmetry in funnel plots or a correlation between study size and effect size. While these methods cannot directly confirm publication bias, they highlight its possible presence. Some techniques also adjust for potential bias, providing corrected estimates of the overall effect size to improve the reliability of the meta-analysis

The Small-Study Effects Concept

  1. Large Studies: Because they require significant resources, large studies are almost always published, regardless of their results.
  2. Moderate-Sized Studies: These are at some risk of not being published, especially if their results are non-significant. However, moderate statistical power often allows significant findings, increasing their chances of publication.
  3. Small Studies: These are most vulnerable to publication bias. Only small studies showing very large (and often exaggerated) effects tend to be published, while those with non-significant results remain unpublished (“file drawer problem”).

The reasoning is straightforward: Publication bias exists because SIGNIFICANT RESULTS are more likely to be published. Since LARGER studies have a HIGHER probability of producing significant results due to their larger sample sizes, publication bias disproportionately affects SMALL STUDIES, leading to an over-representation of large effect sizes in the published literature.

Studies with small sample sizes ➞ Low precision (high standard errors) ➞ Wider confidence intervals ➞ Greater likelihood of non-significant results.

This is precisely why methods for assessing publication bias focus on the issue of small-study effects. As the name suggests, these approaches are particularly concerned with the unique challenges posed by small studies.

Methods for Assessing Publication Bias - Small-Study Effects Methods

Summary

  1. Funnel Plot and Contour-Enhanced Funnel Plot - QUALITATIVE way to inspect asymmetry.
  2. Tests for asymmetry detection - QUANTITATIVE way to assess asymmetry.

Funnel Plot

It is common practice to assess small-study effects using funnel plots. A funnel plot is a scatter plot that displays:

  1. X-axis: the observed effect sizes of studies;
  2. Y-axis: the standard error or precision (1/se) of the observed effect sizes of studies. Typically, the y-axis is inverted in funnel plots, so higher positions on the plot correspond to lower standard errors, indicating more precise estimates;
  3. The vertical line in the middle of the funnel: represents the pooled effect estimate (either in random-effects or fixed-effects models);
  4. The plot also includes the idealized funnel shape that represents the expected distribution of studies in the absence of bias.

Take a look at the example below:

In the absence of small-study effects (no publication bias), the studies should approximately follow the idealized funnel shape displayed in the plot. Is this true in our example? Subjectively, we can say that the studies do follow the shape reasonably well.

  • Studies located in the top part of the plot have low SE and should lie closely together, and not far away from the pooled effect size.

  • Studies located in the lower part of the plot, with increasing SE, the funnel “opens up”, and effect sizes are expected to scatter more heavily to the left and right of the pooled effect.

In this example there are 2 small studies with very low effect sizes to “balance out” the one with very high effects.

Now, let’s explore an example of funnel plot asymmetry:

In this case, the studies deviate from the idealized symmetrical funnel shape, suggesting the presence of small-study effects or potential publication bias.

  • There are three small studies with very high effect sizes located in the bottom-right corner of the plot (‘Shapiro’, ‘Kang’, and ‘Danitz-Orsillo’).

  • However, there are no corresponding small studies in the bottom-left corner of the plot with low or negative effect sizes to “balance out” these high-effect studies.

  • This imbalance contributes to the asymmetry in the funnel plot and may indicate the presence of small-study effects or publication bias.

  • Another concerning observation is that the study with the greatest precision in our sample, conducted by de ‘Vibe’, does not align well with the expected funnel pattern. Its effect size is noticeably smaller than anticipated for a study with such high precision.

Overall, the dataset exhibits an asymmetrical pattern in the funnel plot, which could suggest publication bias. It is possible that the three small studies with high effect sizes represent outliers that happened to achieve significant results, while a hidden pool of unpublished studies with similar standard errors but smaller, non-significant effects remains unaccounted for. This imbalance highlights the potential for selective reporting to skew the overall findings.

Contour-Enhanced Funnel Plot

A good way to inspect how asymmetry patterns relate to statistical significance is to generate contour-enhanced funnel plots (Peterset al., 2008). Such plots can help to distinguish publication bias from other forms of asymmetry.

Usually, the statistical significance thresholds are 0.95 and 0.99, which equals p < 0.05 and p < 0.01, respectively. We see that the funnel plot now contains three shaded regions. We are particularly interested in the p ≥ 0.05, p < 0.05, and p < 0.01 regions.

In the funnel plot, the standard error of the treatment estimates is plotted on the y-axis by default which is likely to be the best choice (Example A). However, we also can plot on the y-axis the 1/SE (precision), as demonstrated in Example B.

Example A

Example B

Example of funnel plot asymmetry PROBABLY due to PUBLICATION BIAS:

Studies appear to be MISSING in areas of statistical NON-SIGNIFICANT or SIGNIFICANT NEGATIVE/LOWER effect (flattened in orange) ➤ Increase the possibility of that the asymmetry is due to publication bias.

See the example below:

The green ellipse indicates likely areas where ‘‘missing’’ studies are expected.
The green ellipse indicates likely areas where ‘‘missing’’ studies are expected.
Example of funnel plot asymmetry PROBABLY due to OTHER SOURCES:

Studies appear to be MISSING in areas of HIGHER STATISTICAL SIGNIFICANCE (flattened in pink) ➤ Reduces the plausibility that publication bias.

See the example below:

Look after other sources of ‘begin’ funnel plot asymmetry:

  • Between-study heterogeneity;

  • Study procedures were different in small studies;

  • High risk-of-bias - Low quality studies;

  • Random (by chance).

Creating a Funnel Plot

We will now make this more concrete by generating a funnel plot our selves. In the meta package, the funnel() function can be used to print a funnel plot for a meta-analysis object. Here, we produce a funnel plot for our metabin() meta-analysis object.

Load Packages

library(meta)
library(readxl)
library(dmetar)

Importing the Dataset

##                   Author Year event.e   n.e event.c  n.c  p2y12_type
## 1  Almendro-Delia et al. 2015     188  3973     116 2076 Clopidogrel
## 2         ATLANTIC Trial 2014      30   906      19  952  Ticagrelor
## 3         Danchin et al. 2018       9   360      18  360 Clopidogrel
## 4          Load&Go Trial 2013       2    56       0   56 Clopidogrel
## 5           CIPAMI Trial 2012       1   164       4  171 Clopidogrel
## 6          Fabris et al. 2022      17   237      17  300 Clopidogrel
## 7          Postma et al. 2014      46  1345      35 1096 Clopidogrel
## 8          Dorler et al. 2011      55  1635     110 1076 Clopidogrel
## 9            Lupi et al. 2016       5   143      13  143  Ticagrelor
## 10        Redfors et al. 2019    1960 37840     528 6964 Clopidogrel
## 11           Yudi et al. 2018      25   892      96 1915 Clopidogrel
## 12      Alexander et al. 2008      12  1029      55 2756  Ticagrelor
## 13          Fefer et al. 2009      12   217       6  166 Clopidogrel
## 14            Lev et al. 2008       1   165       2  127 Clopidogrel
##      overal_risk_of_bias pretreatment _timing %_HTN
## 1   Serious-risk of Bias         Pre-hospital  30.0
## 2       Low-risk of Bias         Pre-hospital  28.8
## 3       Low-risk of Bias         Pre-hospital  25.9
## 4  Moderate-risk of Bias         Pre-hospital  52.3
## 5       Low-risk of Bias         Pre-hospital  49.8
## 6  Moderate-risk of Bias         Pre-hospital  30.5
## 7   Serious-risk of Bias         Pre-hospital  20.3
## 8  Moderate-risk of Bias         Pre-hospital  40.1
## 9   Serious-risk of Bias         Pre-hospital  60.6
## 10 Moderate-risk of Bias         Pre-hospital  40.5
## 11 Moderate-risk of Bias         Pre-hospital  43.4
## 12  Serious-risk of Bias          In-hospital  61.4
## 13  Serious-risk of Bias          In-hospital  32.6
## 14  Serious-risk of Bias          In-hospital  70.1

Performing the Meta-Analysis

m.acm <- metabin(event.e, n.e, event.c, n.c,
                 data = ma$acm,
                 method = "MH",
                 method.tau = "DL",
                 sm = "RR",
                 studlab = Author)

summary(m.acm)
##                           RR             95%-CI %W(common) %W(random)
## Almendro-Delia et al. 0.8469 [0.6761;   1.0608]       11.0       13.0
## ATLANTIC Trial        1.6591 [0.9407;   2.9262]        1.3        8.3
## Danchin et al.        0.5000 [0.2277;   1.0981]        1.3        5.9
## Load&Go Trial         5.0000 [0.2455; 101.8321]        0.0        0.6
## CIPAMI Trial          0.2607 [0.0294;   2.3078]        0.3        1.2
## Fabris et al.         1.2658 [0.6606;   2.4255]        1.1        7.3
## Postma et al.         1.0710 [0.6951;   1.6502]        2.8       10.1
## Dorler et al.         0.3291 [0.2403;   0.4506]        9.6       11.8
## Lupi et al.           0.3846 [0.1408;   1.0508]        0.9        4.3
## Redfors et al.        0.6832 [0.6227;   0.7495]       64.4       14.2
## Yudi et al.           0.5591 [0.3626;   0.8619]        4.4       10.1
## Alexander et al.      0.5844 [0.3142;   1.0867]        2.2        7.6
## Fefer et al.          1.5300 [0.5864;   3.9914]        0.5        4.6
## Lev et al.            0.3848 [0.0353;   4.1970]        0.2        1.0
## 
## Number of studies: k = 14
## Number of observations: o = 67120 (o.e = 48962, o.c = 18158)
## Number of events: e = 3382
## 
##                          RR           95%-CI     z  p-value
## Common effect model  0.6887 [0.6380; 0.7433] -9.57 < 0.0001
## Random effects model 0.7274 [0.5673; 0.9327] -2.51   0.0121
## 
## Quantifying heterogeneity:
##  tau^2 = 0.1108; tau = 0.3328; I^2 = 73.7% [55.4%; 84.5%]; H = 1.95 [1.50; 2.54]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  49.46   13 < 0.0001
## 
## Details on meta-analytical method:
## - Mantel-Haenszel method (common effect model)
## - Inverse variance method (random effects model)
## - DerSimonian-Laird estimator for tau^2
## - Mantel-Haenszel estimator used in calculation of Q and tau^2 (like RevMan 5)
## - Continuity correction of 0.5 in studies with zero cell frequencies

Creating the Funnel Plot

Classic Funnel Plot
funnel(m.acm,
       studlab = TRUE, 
       bg = "lightblue", 
       cex = 2,
       cex.studlab = 1,
       random = T,
       fixed = F,
       backtransf = F)

title("Funnel Plot for All-Cause Mortality")

Contour-Enhanced Funnel Plot
# Define fill colors for contour

funnel(m.acm,
       studlab = TRUE, 
       bg = "lightblue", 
       cex = 2,
       cex.studlab = 1,
       random = T,
       fixed = F,
       backtransf = F,
       contour = c(0.95, 0.99))

legend(x = 1.65, y = 0.01,
       legend = c("p ≥ 0.05", "p < 0.05", "p < 0.01"),
       fill = c("white" , "#808080", "#E6E6E6"))

title("Contour-Enhanced Funnel Plot for All-Cause Mortality")

Tests for Asymmetry Detection in the Funnel Plots

Tests for funnel plot asymmetry evaluate whether there is a systematic relationship between intervention effects and study size beyond what would occur by chance, indicating potential small-study effects.

While visual inspection of contour-enhanced funnel plots can highlight “red flags” for publication bias, such as asymmetry, this approach is subjective and lacks explicit rules for defining “too asymmetric.”

Quantitative tests provide a more objective assessment, complementing visual interpretation.

General Recommendations for Applying Tests for Asymmetry Detection

  • Tests for funnel plot asymmetry should only be used when a meta-analysis includes at least 10 studies, as fewer studies lack sufficient power to distinguish random variation from true asymmetry.

  • These tests are also inappropriate when all studies are of similar sizes (i.e., similar standard errors), as this limits their ability to detect asymmetry.

  • When small-study effects are identified, publication bias should be considered one of several possible explanations. While funnel plots and asymmetry tests highlight potential issues, they do not offer solutions.

  • Review authors should note that these tests have low power, so a lack of evidence for asymmetry does not rule out the presence of bias, including publication bias.

Which method to choose?

Scenarios Methods R Syntax
Mean Differences Egger 1997 "Egger"
Odds Ratios Harbord 2006 or Peters 2006 "Harbord" | "Peters"

Risk Ratios

Risk Differences

 Egger 1997 or Peters 2006 "Egger" | "Peters"
Standardized Mean Differences Pustejovsky & Rodgers 2019 "Pustejovsky"
Meta-analysis of diagnostic test accuracy studies Deeks 2005 "Deeks"

How to perform the tests for asymmetry detection

To apply the tests to detect asymmetry we will use the metabias() function from the meta package. We need to provide some instructions for the metabias() function.

metabias(m.acm,
         method.bias = "Egger",
         plotit = T)

## Linear regression test of funnel plot asymmetry
## 
## Test result: t = 0.34, df = 12, p-value = 0.7363
## Bias estimate: 0.2498 (SE = 0.7246)
## 
## Details:
## - multiplicative residual heterogeneity variance (tau^2 = 4.0812)
## - predictor: standard error
## - weight:    inverse variance
## - reference: Egger et al. (1997), BMJ

After running the metabias() function, the next step is to annotate the contour-enhanced funnel plot with the results from the test used to assess symmetry. Here’s how to proceed:

funnel(m.acm,
       studlab = TRUE, 
       bg = "lightblue", 
       cex = 2,
       cex.studlab = 1,
       random = T,
       fixed = F,
       backtransf = F,
       contour = c(0.95, 0.99))

title("Contour-Enhanced Funnel Plot for All-Cause Mortality")

legend(x = 1.65, y = 0.01,
       legend = c("p ≥ 0.05", "p < 0.05", "p < 0.01"),
       fill = c("white" , "#808080", "#E6E6E6"))

legend(x = -2.9, y = 1.39, bty = "n", cex = 0.8,
       legend = c(expression(bold("p-val = 0.74 (Egger's Test)")), "intercept: 0.25 (SE = 0.7246) | t = 0.34"))

How to interpret?

  1. Look at the Intercept:

    • The intercept (bias estimate) represents the expected standardized effect size (z -score) when study precision is zero (i.e., infinite standard error).

    • A value close to zero suggests symmetry in the funnel plot, indicating no small-study effects or publication bias.

  2. Evaluate the Significance of the Intercept:
    • In this example, the intercept is 0.25, which is very close to zero. • The statistical test (t = 0.34, p = 0.74) shows that the intercept is not significantly different from zero (p > 0.05).
  3. What Does This Mean?

    • Since the intercept is not significantly different from zero, the funnel plot is likely symmetrical.

    • Symmetry in the funnel plot suggests that small-study effects or publication bias are unlikely to have distorted the results.

Duval & Tweedie trim and fill method

data(Fleiss1993bin)

m1 <- metabin(d.asp, n.asp, d.plac, n.plac, data = Fleiss1993bin, sm = "OR")

m1
## Number of studies: k = 7
## Number of observations: o = 28003 (o.e = 14186, o.c = 13817)
## Number of events: e = 4414
## 
##                          OR           95%-CI     z p-value
## Common effect model  0.8969 [0.8405; 0.9570] -3.29  0.0010
## Random effects model 0.8683 [0.7559; 0.9973] -2.00  0.0457
## 
## Quantifying heterogeneity:
##  tau^2 = 0.0147 [0.0000; 0.1145]; tau = 0.1214 [0.0000; 0.3384]
##  I^2 = 39.7% [0.0%; 74.6%]; H = 1.29 [1.00; 1.99]
## 
## Test of heterogeneity:
##     Q d.f. p-value
##  9.95    6  0.1269
## 
## Details on meta-analytical method:
## - Mantel-Haenszel method (common effect model)
## - Inverse variance method (random effects model)
## - Restricted maximum-likelihood estimator for tau^2
## - Q-Profile method for confidence interval of tau^2 and tau
forest(m1,
       layout = "Revman5",
       sortvar = TE,
       lab.e = "Experimental", label.left = "Favors Experimental",
       lab.c = "Control", label.right = "Favors Control",
       ff.lr = "bold",
       leftcols = c("studlab","event.e", "n.e", "event.c", "n.c", "w.random", "effect", "ci" ) ,
       leftlabs = c("Studies", NA, NA, NA, NA, NA, NA, NA),
       text.random = "Random effects model",
       random = TRUE, 
       common = FALSE,
       test.overall.random = TRUE,
       rightcols = FALSE,
       colgap = "3mm",
       fs.heading = 12,
       fs.study = 12, 
       fs.hetstat = 10, 
       digits = 2,
       digits.pval = 2,
       pooled.events = TRUE,
       pooled.totals = TRUE,
       print.I2.ci = TRUE,
       col.square="darkcyan", col.square.lines="black",
       prediction = TRUE, col.predict = "#CEF2EE",col.predict.lines = "black", ff.predict = 1)

tf1<-trimfill(m1)

summary(tf1)
##               OR           95%-CI %W(random)
## 1         0.7197 [0.4890; 1.0593]        6.8
## 2         0.6808 [0.4574; 1.0132]        6.5
## 3         0.8029 [0.6065; 1.0629]       10.8
## 4         0.8007 [0.4863; 1.3186]        4.5
## 5         0.7981 [0.5526; 1.1529]        7.4
## 6         1.1327 [0.9347; 1.3728]       16.4
## 7         0.8950 [0.8294; 0.9657]       26.8
## Filled: 5 1.0466 [0.7246; 1.5118]        7.4
## Filled: 1 1.1607 [0.7886; 1.7084]        6.8
## Filled: 2 1.2271 [0.8244; 1.8263]        6.5
## 
## Number of studies: k = 10 (with 3 added studies)
## Number of observations: o = 31987 (o.e = 16369, o.c = 15618)
## Number of events: e = 4775
## 
##                          OR           95%-CI     z p-value
## Random effects model 0.9228 [0.8228; 1.0350] -1.37  0.1699
## 
## Quantifying heterogeneity:
##  tau^2 = 0.0113 [0.0000; 0.1145]; tau = 0.1061 [0.0000; 0.3384]
##  I^2 = 37.4% [0.0%; 70.1%]; H = 1.26 [1.00; 1.83]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  14.37    9  0.1099
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-Profile method for confidence interval of tau^2 and tau
## - Trim-and-fill method to adjust for funnel plot asymmetry (L-estimator)
funnel(tf1,
       studlab = TRUE, 
       bg = "lightblue", 
       cex = 2,
       cex.studlab = 1,
       random = T,
       fixed = F,
       backtransf = F,
       contour = c(0.95, 0.99))

forest(tf1,
       layout = "Revman5",
       sortvar = TE,
       lab.e = "Experimental", label.left = "Favors Experimental",
       lab.c = "Control", label.right = "Favors Control",
       ff.lr = "bold",
       leftcols = c("studlab","event.e", "n.e", "event.c", "n.c", "w.random", "effect", "ci" ) ,
       leftlabs = c("Studies", NA, NA, NA, NA, NA, NA, NA),
       text.random = "Random effects model",
       random = TRUE, 
       common = FALSE,
       test.overall.random = TRUE,
       rightcols = FALSE,
       colgap = "3mm",
       fs.heading = 12,
       fs.study = 12, 
       fs.hetstat = 10, 
       digits = 2,
       digits.pval = 2,
       pooled.events = TRUE,
       pooled.totals = TRUE,
       print.I2.ci = TRUE,
       col.square="darkcyan", col.square.lines="black",
       prediction = TRUE, col.predict = "#CEF2EE",col.predict.lines = "black", ff.predict = 1)

How to report?

For regression test results, it is usually sufficient to report the value of the intercept, its standard error, as well as the 𝑡 and 𝑝 value. It is common practice to include them alongside visual representations, such as a funnel plot, to provide a clearer context for interpretation.

In the methods section:

Examples:

  • “The presence of small-study effects was assessed by visual assessment of the contour-enhanced funnel plots and formal Egger regression test.1-3”

  • “We explored the potential for publication bias by visual inspection of the contour-enhanced funnel plot and Peters’ test.1-3”

In the results section:

Examples:

  • “There was no evidence of small-study effects by visual assessment of the contour-enhanced funnel plots or by Egger regression test (see Supplementary Figure X).”

  • “Contour-enhanced funnel plot analysis and Egger regression test detected evidence of publication bias for the primary endpoint (intercept = 1.25, SE = 0.52, t = 2.40 , p = 0.001 ), suggesting potential small-study effects or selective reporting (see Supplementary Figure X).”

  • “We did not assess publication bias due to the limited number of studies included (n < 10), as regression-based methods lack sufficient power to distinguish chance from true asymmetry.”

References:

  1. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61(10):991-6.

  2. Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, Carpenter J, Rücker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JP. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011 Jul 22;343:d4002.

  3. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997 Sep 13;315(7109):629-34.

  4. Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006 Oct 30;25(20):3443-57.

  5. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006 Feb 8;295(6):676-80.

  6. Pustejovsky JE, Rodgers MA. Testing for funnel plot asymmetry of standardized mean differences. Res Syn Meth. 2019; 10: 57–71.

  7. Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol. 2005 Sep;58(9):882-93.